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student listening circle encourages students to share with adults their 
experiences and ideas on an important school topic and to collaborate 
in planning and implementing school improvement practices. This 
randomized controlled trial is the first to examine whether participation 
in a listening circle affects students’ perceptions of their influence on 
school improvement efforts; their sense of being connected to school 
staff, peers, and the school as a whole; their perceived competency 
for improving the school; and their sense of academic self-efficacy. The 
study found no discernible impacts on participating students. Descriptive 
analyses of school staff perceptions did find that a larger proportion of 
staff reported after the intervention than before it that students had the 
ability to make meaningful contributions to school improvement. However, 
the descriptive findings on staff perceptions do not support conclusions 
about the impact of the intervention. 
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Summary 


An activity for eliciting student involvement in collaborative decisionmaking and problem- 
solving with adults — the student listening circle workshop — is examined for the first time 
through an experimental study of its effects on participating students. A student listening 
circle is a facilitated focus group in which students articulate to adults their experienc- 
es, perspectives, and ideas on an important school topic and then collaborate with those 
adults to plan and implement related actions to improve their school climate. 

Although the student listening circle is intended partly as a tool to gather data to be used 
in school improvement efforts, it can also be considered a student intervention (O’Mal- 
ley, Voight, & Izu, 2013). Accordingly, this study examines the potential impact of stu- 
dents’ participation in listening circles on the students involved. Conducted by the School 
Climate Alliance with technical support from Regional Educational Laboratory (REL) 
West, the study took place at nine middle schools in eight California school districts in 
spring 2015. 

The study has a primary experimental component and a secondary descriptive component. 
The primary component is a random assignment investigation of the impacts of student 
listening circles on the perceptions of student participants. The secondary component is 
a descriptive study of staff participants’ perceptions before and after a student listening 
circle. The primary component used surveys to ascertain student perceptions of school 
climate and of their competencies and compared responses from an intervention group 
of students with those from a control group 1 week before and 1 week and 12 weeks after 
participation in a student listening circle. The secondary component used surveys to assess 
changes in staff participants’ perceptions of school supports and of student competencies 
after the student listening circles, as well as interviews to assess staff perceptions of practic- 
es implemented as a result of the intervention. 

The experimental results showed no discernible effects of the student listening circle on 
student participants’ perceptions of school climate or on their competencies. Participation 
in the student listening circle was not associated with changes in student perceptions 
about their input into decisionmaking at school, relationships with school staff and peers, 
school bonding (sense of connectedness/belonging at school), competencies for improving 
the school (students’ perceived ability to effect school change), or academic self-efficacy 
(perceived ability to succeed academically). 

The descriptive results of the study show that after participating in a student listening 
circle, a larger percentage of school staff reported the belief that students have opportuni- 
ties for meaningful participation at school, trust in students, and recognition of students’ 
competency in school improvement. These descriptive results do not provide evidence 
about the impacts of student listening circles because there was no staff control group. 
Thus any increases in staff perceptions could be due to factors other than student partici- 
pation in a listening circle. Moreover, students’ actual opportunities and competencies were 
not directly measured in the study — only staff and student perceptions of opportunities. 

According to interviews with school principals and student listening circle coordinators, 
schools followed through with most of the actions suggested during the student listening 


circles and implemented multiple school-improvement practices to address issues identified 
during the student listening circles. 

Although the experimental findings suggest that the student listening circle has no dis- 
cernible impact on student participants, there are other reasons to implement and conduct 
further research. Student listening circles are also intended to improve the overall school 
climate by altering perceptions of staff, actively promoting a more positive school climate, 
and implementing schoolwide practices. Future studies with a different design may seek to 
ascertain the extent to which student listening circles have schoolwide effects other than 
on the perceptions of student and staff participants. Potential schoolwide impacts include 
effects on decisionmaking practices in schools, school bonding, and improved relation- 
ships between school staff and students. 
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Why this study? 


Performance-based accountability systems that hold teachers and administrators respon- 
sible for student achievement (Hanson, Austin, & Lee-Bayha, 2004) rarely offer students 
a voice in education practices and school reform (Cook-Sather, 2002; Joselowsky, 2007; 
Mitra, 2006, 2008a, 2008b). Increasingly, however, school improvement initiatives are 
including efforts to elicit and incorporate student perspectives on teaching and learn- 
ing (Harris et al., 2014) because of a growing recognition that students can contribute to 
school improvement (Education Alliance, 2004). 

Input from students involved in collaborative decisionmaking and problem-solving with 
adults is called “student voice” (Camino, 2000; O’Donoghue, Kirshner, & McLaughlin, 
2002). Promoting student voice is motivated not only by the notion that reinforcing dem- 
ocratic values and capacities is a key mission of public education and should therefore be 
modeled in school decisionmaking but also by the notion that students’ unique experi- 
ences and perspectives on schooling can be used to advance school climate improvement 
efforts. 

Giving students the opportunity to tackle issues important to them, in partnership with 
supportive adults, can be an instrument of individual and organizational growth. Including 
student voice in school improvement initiatives has been associated with enhancements 
in teaching, learning, and teacher-student relationships (Fielding, 2001; Mitra, 2003; Soo 
Hoo, 1993). Students who contribute to school improvement tend to enjoy better rela- 
tionships with teachers and higher levels of academic motivation (Ames, 1992; Eccles, 
Wigfield, & Schiefele, 1998; Lee & Zimmerman, 1999). Students with responsibility for 
and some control over the development of school policies and practices appear to be more 
likely to take ownership of education goals and to become invested in academic and other 
types of success (Black & Deci, 2000; Newman, 1992). 

Growing evidence suggests that students, working together and with adults, can effect 
change in the school climate (Christens & Kirshner, 2011; Shah & Mediratta, 2008). 
Research and theory on youth civic engagement identify three potential pathways for using 
student contributions to change school climate: collaborating on direct action to change 
policy and practice, strengthening relationships among students and between students and 
staff, and expanding students’ social and emotional competence (Voight, 2012a; 2012b). 

All three pathways could be accessed through a school-based student voice initiative. 
Students with a voice in school decisionmaking may be able to leverage policy changes, 
strengthen peer and staff-student social networks, and develop their own socioemotional 
competencies. These outcomes can improve school climate. While qualitative case-study 
research has built empirical support for these connections, there is scant quantitative or 
experimental evidence on the effects of student voice initiatives. 

This study examines the impact of one activity for eliciting student voice — the student 
listening circle. This activity fosters collaboration between students and adults to empower 
students and improve the school climate (Benard & Slade, 2009; Burgoa & Izu, 2010). 

The 16 schools and 11 school districts in the School Climate Alliance, as well as the Cal- 
ifornia Department of Education, have been interested in implementing student listening 
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circles. The School Climate Alliance is a networked school improvement community in 
which school teams analyze school safety and climate data to select, implement, and test 
appropriate interventions. In 2011/12 the California Department of Education’s Coordinat- 
ed School Health and Safety Office sponsored interventions — including listening circles 
— in the 58 comprehensive high schools with the greatest school climate improvement 
needs under a grant from the federal Safe and Supportive Schools program. 1 The School 
Climate Alliance and the California Department of Education thus have a strong interest 
in learning about the effectiveness of student listening circles. 


The School Climate Alliance, with support from Regional Educational Laboratory (REL) 
West, conducted this study in nine middle schools in eight California school districts in 


2015. 


What the study examined 


This study is the first to rigorously examine the short-term effects of student listen- 
ing circles on students’ perceptions of school climate and their own competencies. This 
short-cycle experimental trial also demonstrates how to rigorously investigate the effective- 
ness of student listening circles. 

The theory of action behind student listening circles hypothesizes an influence on the 
overall school climate through direct impacts on school practices, including schoolwide 
governance, that contribute to school climate and through the perceptions of adults 
and students who participate in the student listening circle as well as those who do not 
(figure 1). Though designed partly to be a data-gathering tool, the student listening circle 
can itself be considered an intervention (O’Malley et al, 2013). 


This study is the 
first to rigorously 
examine the short- 
term effects of 
student listening 
circles on students’ 
perceptions of 
school climate 
and their own 
competencies 


The student listening circle was developed by Benard (2004) and Benard and Slade (2009) 
to apply insights from resilience research to school settings (box 1). It was refined through 
its use in 58 California high schools. 

This study applied experimental methods to investigate just one aspect of the theory of 
action underlying the student listening circle — the short-term impacts of participation 
on students’ perceptions (figure 1, shaded boxes). It looked at whether participation in a 
student listening circle affected students’ perception of the school climate and perceptions 
of their own ability to influence the school climate and of their academic ability. It is pos- 
sible for the student listening circle to have no discernible impacts on student participants 
but still to have impacts on school climate. Descriptive analysis was also used to examine 
staff participants’ perceptions before and after the student listening circle. 


Experimental and descriptive components of the study 

This study had a primary experimental component and a secondary descriptive compo- 
nent. The primary component was a random assignment investigation of the impacts of 
student listening circles on student participants. The secondary component was a descrip- 
tive analysis of staff participants’ perceptions before and after the student listening circle. 
The study also describes how student listening circles were conducted in study schools (see 
box 1). Data sources and methods are described in box 2. 
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Figure 1. Theory of action behind student listening circles 



Note: Shaded boxes and solid arrows indicate the components of the action model that were rigorously ex- 
amined using an experimental design. Dashed arrows indicate relationships that were not examined using an 
experimental design. 

Source: Authors' construction based on O’Malley et al. (2013). 


Box 1. The student listening circle 

A student listening circle is a facilitated focus group in which students share with adults their 
experiences and ideas on an important school-related topic and then collaborate with those 
adults to plan and implement actions aimed at school improvement. Ideally, a student listening 
circle is composed of 8-10 students and no fewer than twice as many (16-20) adults (O’Mal- 
ley et al., 2013). (On average, 10 students and 16 adults participated in the student listening 
circles conducted for this study. Each student listening circle included students from differ- 
ent grades, male and female students, and students with varied academic performance. The 
adults included school staff members, parents, and community representatives.) 

Most of the process takes place at the school in a workshop lasting several hours. For 
the adults the process starts with a 20-minute orientation where participants are asked to 
abide by agreements designed to make the process feel safe for students, such as turning off 
cell phones, being silent, not attributing student comments to individual students outside the 
listening circle activities, and not approaching students for clarification outside of preplanned, 
structured follow-up activities. 

For the students the process starts with a 60-minute practice session during which a 
trained facilitator — a school staff member or someone from outside the school — asks 

(continued) 
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Box 1. The student listening circle (continued) 


questions, actively listens to students’ responses, reflects their answers, and shows students 
how to reframe their responses in constructive, solution-focused ways. The questions, chosen 
by facilitators and school leaders, are intended to elicit information about an area of interest 
for the school about which the school needs more information to improve decisionmaking. Like 
adults, students are asked to agree to turn off cell phones during the listening circle, provide 
productive input, be respectful, use staff or student names only when making positive com- 
ments, speak one at a time, and be mindful of time. 

In the 70-minute listening circle students speak to the adults, addressing four open-ended 
questions on the given topic (see below). Students and the facilitator sit facing one another in 
a circle, while adults sit in a circle around the student circle. As the students speak, the adults 
listen carefully. Sample student listening circle discussion questions are listed here (questions 
with asterisks were used in this study): 

• How do you know when an adult at school cares about you? What do they say and do?* * 

• How do you know when an adult at school believes in you? What do they say and do? 

• What makes your classes engaging or interesting to you? What could teachers do to make 
your classes more engaging and interesting?* 

• What kinds of things could the school do that would make it a better place to learn and 
make friends? What could students do to help?* 

• If you could change or improve one thing at school, what would it be?* 

• What could you do at your school that would make a difference? 

• What are your hopes and dreams, and how can adults at school help you achieve them? 
After students share their responses, both students and adults reflect on the process. 

The listening circle is followed by a 70-minute facilitated dialogue in which student and 

adult participants collaborate in planning and assigning responsibility for actions aimed at 
school improvement. Participants develop a list of short- and long-term actions, including at 
least one short-term action that the participants agree to perform within 30 days. 

In subsequent days and weeks the facilitator works with school administrators and partic- 
ipants in student listening circles to review and revise the list of actions. Then the facilitator 
disseminates notice of the agreed actions schoolwide, and the relevant staff, students, and 
other parties implement them. Typically, one staff member volunteers to oversee the imple- 
mentation of a particular action through consultation and collaboration with students and 
others who participated in the planning dialogue. 


The primary component, an impact study, investigated the following experimental research 
question: 

• What is the short-term impact of participation in student listening circles on 
student perceptions of their input into decisionmaking at school, their relation- 
ships with school staff and peers, their bonding to the school, their competency 
for improving the school, and their academic self-efficacy? 

The secondary component examined two descriptive research questions: 

• What are staff participants’ and school administrators’ perceptions of their rela- 
tionships with students, student opportunities for meaningful participation at 
school, student competency for improving the school, trust in students, and their 
own competency in school improvement efforts during the student listening circle? 

• What school actions, policies, and practices took place during and after imple- 
mentation of the student listening circle? 


4 


One week before and 1 week and 12 weeks after the student listening circle the study team 
distributed the student survey to students participating in a student listening circle and 
to a control group of students not participating and distributed the staff survey to staff 
participants. There was no staff control group (see box 2). Table 1 displays the outcome 
categories, sample questions, and answer scale for the student survey, and table 2 displays 
the same information for the staff survey. 

Appendix A provides details of the research design, sites, data, outcomes, and analytic 
strategy. Appendix B describes goals set and actions taken as a result of the student lis- 
tening circle. Ancillary analyses of the student surveys are shown in appendix C, and the 
student and staff surveys are reproduced in appendix D. 


What the study found 


This section presents the study’s findings on the experimental analysis of the short-term 
impacts of student listening circles on student perceptions of school climate and their own 


Table 1. Student survey outcome measures, sample questions, and answer scale, 
2015 



Answer scale 

Student outcome measure and sample questions 

Minimum 

Maximum 

Student school climate perceptions 

Student participation in decisionmaking 

1 = 

4 = 

In our school, students are given a chance to help make decisions. 

strongly 

strongly 

Students in this school have a say in how things work. 

disagree 

agree 

School staff who participated in a student listening circle know students' 

0 = 

1 = 

names 

1 am certain that Mr./Ms. knows mv name. 

0% of staff 

100% of staff 

Comfort talking with school staff who participated in student listening circle 

Please rate how comfortable vou would be talkine to Mr./Ms. 

1 = 

not at all 

5 = 

one-on-one about things that are bothering you. 

very 

Caring relationships/high expectations from staff at school 



At my school, there is a teacher or some other adult who really cares 

1 = 

4 = 

about me. 

not at all 

very 

At my school, there is a teacher or some other adult who always wants 
me to do my best. 

true 

much true 

School bonding 

1 = 

4 = 

1 feel like a real part of this school. 

strongly 

strongly 

1 wish 1 were in a different school, (reverse coded) 

disagree 

agree 

Peer support 

1 = 

4 = 

Students my age really care about what happens to me. 

not at all 

very much 

Students my age only think bad things about me. (reverse coded) 

true 

true 

Competencies 

Competency to improve the school 

Students like me can really understand what’s going on with my school. 

1 = 

strongly 

disagree 

4 = 

strongly 

agree 

There are plenty of ways for students like me to have a say in what our 

school does. 

Academic self-efficacy 

How well can you study when there are other interesting things to do? 

1 = 

not at all 

5 = 

very well 

How well can you pay attention during every class? 


Source: Authors' compilation. 
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Table 2. Staff survey outcome measures, sample questions, and answer scale, 2015 

Staff measure and sample questions 

Answer scale 

Minimum Maximum 

Student school climate perceptions 

Student opportunities for meaningful participation at school 

1 = 

4 = 

This school encourages opportunities for students to decide things like 

class activities and rules. 

Students are involved in helping solve school problems. 

strongly 

disagree 

strongly 

agree 

Student relationships with staff 

i = 

4 = 

Students share their concerns with me. 

strongly 

strongly 

Students talk with me about their homes and families. 

disagree 

agree 

Competencies 

Recognition of student competencies to improve the school 

1 = 

4 = 

Students have a lot to contribute in school improvement decisions. 

strongly 

strongly 

Students offer constructive feedback about the school environment. 

disagree 

agree 

Trust in students 

1 = 

4 = 

Students in this school can be counted on. 

strongly 

strongly 

Students in this school have to be closely supervised, (reverse coded) 

disagree 

agree 

Staff ability to improve the school 

1 = 

4 = 

1 believe that 1 am empowering students. 

strongly 

strongly 

1 believe that 1 am helping kids become independent learners. 

disagree 

agree 

Source: Authors' compilation. 





Box 2. Data and methods 


For this study, conducted in nine middle schools in eight California school districts in 2015, 
256 students volunteered, an average of 28 students per school. A school staff member 
contacted potential student participants, described the student listening circle and the study 
requirements, and asked students whether they wished to participate. Only students who 
agreed and whose parent provided a signed consent form were allowed to participate. 

Within each school 10 students were randomly selected to participate; the remaining vol- 
unteers served as the control group. So that each student listening circle comprised roughly 
equal numbers of male and female students and of students in each grade, randomization was 
stratified on these dimensions. The school principal or study coordinator recruited staff to par- 
ticipate. On average, 16 adults in each school attended the student listening circle, of which 
11 were school staff and eligible to participate in the study. Appendix A provides details about 
the characteristics of student and staff participants. 

Impacts were estimated by examining differences between intervention and control groups 
on key outcomes. Students who participated in the study (n = 256) and school staff who 
attended a student listening circle (n = 110) completed surveys 1 week before participation 
in the student listening circle (pretest), 1 week after participation (posttest), and 12 weeks 
after participation (follow-up). Survey response rates ranged from 91 percent (posttest) to 
97 percent (pretest) for students and from 59 percent (follow-up) to 79 percent (pretest) for 
eligible staff. Posttest and follow-up staff surveys were not collected from two schools. 

Student survey data were analyzed for results in two outcome domains: school climate 
perceptions and competencies. The study gauged school climate perceptions on the basis of 
students’ perceptions about their participation in decisionmaking, student-staff relationships, 

(continued) 
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Box 2. Data and methods (continued) 


school bonding, and peer support. The study measured personal competency outcomes by 
students’ perceptions of their competency for improving the school and their academic self-ef- 
ficacy (see table 1). In the experimental analyses of student outcomes, multiple comparison 
procedures were used to adjust for the increased probability of finding statistically significant 
differences between the intervention and control groups due to the number of statistical tests 
conducted (see appendix A). Adjusted p-values, which account for multiple significance testing, 
were used to make inferences about program impacts. 

The study measured staff perceptions of student-staff relationships, students’ oppor- 
tunities for meaningful participation at school, students’ competencies for improving the 
school, staff members’ trust in students, and staff members’ ability to improve the school 
(see table 2). The study also included structured interviews with administrators after imple- 
mentation of the student listening circle in order to determine their perceptions of practices 
implemented as a consequence of the student listening circle. 


competencies. It also describes the findings of descriptive analyses of staff perceptions of 
their relationships with students, student involvement and competencies, and their own 
engagement in school improvement efforts over a 13'week period. 

Experimental results indicate that participation in student listening circles had no impact on 
student participants’ perceptions of either school climate or their own competencies 

The random assignment investigation found no statistically significant impact of 
student participation in the student listening circle on students in any of the areas 
assessed 1 week or 12 weeks after the student listening circle took place (tables 3 and 
4). There were no statistically significant postintervention differences between interven- 
tion group and control group students in perceptions of input into decisionmaking at 
school, relationships with school staff and peers, school bonding, ability to contribute to 
school improvement efforts, or academic self-efficacy. 2 


The random 
assignment 
investigation found 
no statistically 
significant 
impact of student 
participation in the 
student listening 
circle on students 
in any of the areas 
assessed 1 week or 
12 weeks after the 
student listening 
circle took place 


Descriptive results showed little change in students’ perceptions, while staff participants saw 
improvements in student competencies and had more trust in students and belief in their abilities 


Most measures of student perceptions of school climate and competencies showed no 
statistically significant increases across time, although students reported that greater 
percentages of staff who participated in student listening circles knew their names. 

Descriptive analyses were conducted to examine the extent to which students’ percep- 
tions changed in both intervention and control groups. The results suggest that neither 
the intervention group nor the control group students’ perceptions of school climate and 
competencies had increased one week after participation in student listening circles (see 
table Cl in appendix C). Twelve weeks after participation, however, students in both 
groups reported that greater percentages of staff who participated in the student listening 
circles knew their names. Students in the intervention group (but not in the control 
group) also reported higher average levels of participation in decisionmaking at school 
and greater levels of comfort talking with staff participants in the student listening circles 
than they had before their student listening circle experience (see table C2 in appendix 
C). The increase in perceptions of decisionmaking opportunities reported by students 
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Table 3. Impact of participation in student listening circle on student perceptions 1 
week after participation, 2015 



Adjusted 

means 

Difference between 
intervention and control groups 


Student outcome measure 

Intervention 

group 

(standard 

deviation) 

Control 

group 

(standard 

deviation) 

Difference 3 

(standard 

error) 

p-value 

(adjusted 

p-value) 

Effect 

size 

Student 

sample 

size 

School climate perceptions 

Student participation in 

2.18 

2.26 

-0.07 

.28 

-0.11 

234 

decisionmaking 

(0.66) 

(0.64) 

(0.07) 

(.83) 

SLC staff 11 knows name 

0.39 

0.38 

0.00 

.93 

0.00 

236 

(0.40) 

(0.38) 

(0.02) 

(.99) 

Comfort talking with SLC staff 11 

2.62 

2.57 

0.05 

.65 

0.04 

232 

(1.07) 

(1.06) 

(0.10) 

(.99) 

Caring relationships 

3.22 

3.23 

-0.01 

.90 

-0.01 

236 

(0.70) 

(0.63) 

(0.07) 

(.99) 

School bonding 

2.87 

2.87 

0.00 

.98 

0.00 

234 

(0.64) 

(0.57) 

(0.05) 

(.98) 

Peer support 

2.50 

2.57 

-0.08 

.25 

-0.12 

236 

(0.65) 

(0.62) 

(0.07) 

(.99) 

Competencies 

Competency in school 

2.83 

2.79 

0.04 

.53 

0.07 

234 

improvement efforts 

(0.56) 

(0.57) 

(0.06) 

(.99) 

Academic self-efficacy 

3.68 

3.68 

0.01 

.92 

0.01 

233 

(0.80) 

(0.79) 

(0.06) 

(.92) 


a. Differences shown in this column may vary from differences calculated from data in the table because of 
rounding. 

b. SLC staff are staff members who participated in the student listening circle. 

Note: Higher numbers indicate more positive perceptions of school climate and competencies (see table 1 for 
descriptions of the student outcome measures). Multiple regression models were run to estimate program im- 
pacts and standard errors. Data are regression-adjusted to account for differences in baseline characteristics. 
The model included baseline measures of the outcomes, gender, grade, school, and randomization strata. In 
no case was a difference between the intervention and control groups found to be statistically significant at 
the adjusted .05 significance level. 

Source: Authors' analysis of primary data collected for the study through surveys. 


in the intervention group was no greater than that reported by students in the control 
group. Thus perceptions of decisionmaking opportunities did not increase more among 
students in the intervention group than among students in the control group. The sta- 
tistically significant increase in intervention students’ comfort talking with staff partici- 
pants in the student listening circles is consistent with the experimental results shown in 
table 4 (see note 2). 3 

In contrast, all students’ average reported levels of competency in school improvement 
efforts and academic self-efficacy declined between the pretest and the 12-week follow-up. No 
other changes in school climate were statistically significant (see table C2 in appendix C). 

Staff participants in student listening circles perceived improvements in some student 
competencies. It is not possible to ascertain how staff participants’ perceptions were affect- 
ed by the listening circles because there was no staff control group. However, descriptive 
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Table 4. Impact of participation in student listening circle on student perceptions 
12 weeks after participation, 2015 



Adjusted 

means 

Difference between 
intervention and control groups 


Student outcome measure 

Intervention 

group 

(standard 

deviation) 

Control 

group 

(standard 

deviation) 

Difference 3 

(standard 

error) 

p-value 

(adjusted 

p-value) 

Effect 

size 

Student 

sample 

size 

School climate perceptions 

Student participation in 

2.28 

2.17 

0.11 

.11 

0.18 

236 

decisionmaking 

(0.67) 

(0.63) 

(0.07) 

(.22) 

SLC staff 11 knows name 

0.48 

0.45 

0.02 

.29 

0.09 

235 

(0.28) 

(0.27) 

(0.02) 

(.44) 

Comfort talking with SLC staff 11 

2.75 

2.49 

0.26 

.02 

0.23 

233 

(1.13) 

(1.11) 

(0.11) 

(.11) 

Caring relationships 

3.15 

3.19 

-0.04 

.60 

-0.06 

237 

(0.78) 

(0.69) 

(0.08) 

(.72) 

School bonding 

2.89 

2.86 

0.02 

.64 

0.04 

237 

(0.58) 

(0.56) 

(0.05) 

(.64) 

Peer support 

2.46 

2.56 

-0.10 

.11 

-0.17 

236 

(0.66) 

(0.61) 

(0.06) 

(.33) 

Competencies 

Competency in school 

2.75 

2.76 

-0.01 

.87 

-0.02 

236 

improvement efforts 

(0.62) 

(0.58) 

(0.06) 

(.87) 

Academic self-efficacy 

3.58 

3.66 

-0.08 

.29 

-0.09 

236 

(0.81) 

(0.85) 

(0.07) 

(.58) 


a. Differences shown in this column may vary from differences calculated from data in the table because of 
rounding. 

b. SLC staff are staff members who participated in the student listening circle. 

Note: Higher numbers indicate more positive perceptions of school climate and competencies (see table 1 for 
descriptions of the student outcome measures). Multiple regression models were run to estimate program im- 
pacts and standard errors. Data are regression-adjusted to account for differences in baseline characteristics. 
The model included baseline measures of the outcomes, gender, grade, school, and randomization strata. In 
no case was a difference between the intervention and control groups found to be statistically significant at 
the .05 adjusted significance level. 

Source: Authors' analysis of primary data collected for the study through surveys. 


analyses indicate a change over time in staff participants’ perceptions. Specifically, after 
the intervention staff reported higher average levels of opportunities for students to partic- 
ipate meaningfully at school, student competency for school improvement, and staff trust 
in students. 

Based on the average responses of the 68 school staff who completed the pretest and 
posttest surveys, the only statistically significant differences were increases in staff per- 
ceptions of student competency for improving the school and in their trust in students 
one week after participation in the student listening circle compared with responses prior 
to participation (table 5). The percentage of staff reporting that they agreed or strongly 
agreed that students offer constructive feedback about their school environment (com- 
petency in school improvement efforts) increased from 59 percent before the student lis- 
tening circle to 75 percent one week after (not shown in tables). Similarly, the percentage 
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Table 5. Results of the staff surveys 1 week before and 1 week after participation 
in student listening circle, 2015 



Means 

Difference between 
pretest and posttest 


Staff outcome measure 

Pretest 

(standard 

deviation) 

Posttest 

(standard 

deviation) 

Difference 3 

(standard 

error) 

p-value 

Standardized 

difference 

Staff 

sample 

size 

Belief that students have 

opportunities for meaningful 
participation at school 

2.73 

2.79 

0.07 

.11 

0.12 

68 

(0.55) 

(0.64) 

(0.04) 

Student-staff relationships 

3.15 

3.21 

0.06 

.15 

0.13 

68 

(0.42) 

(0.43) 

(0.04) 

Recognition of student 
competency to improve the 

school 

2.67 

2.91 

0.24* 

.00 

0.50 

68 

(0.48) 

(0.48) 

(0.06) 

Trust in students 

1.29 

1.44 

0.15* 

.03 

0.22 

68 

(0.69) 

(0.39) 

(0.07) 

Staff ability to improve the 

3.44 

3.45 

0.01 

.80 

0.03 

68 

school 

(0.40) 

(0.46) 

(0.04) 


* Statistically different from zero at the p < .05 level based on a two-tailed test. 

a. Differences shown in this column may vary from differences calculated from data in the table because of 
rounding. 

Note: Higher numbers indicate more positive perceptions of school climate and competencies (see table 2 for 
descriptions of the staff outcome measures). P-values are based on a regression model that accounted for 
differences across schools and the clustering of observations within schools. To assess changes in staff-re- 
ported outcomes between pretest and posttest, difference scores were computed by subtracting the pretest 
survey measure from the posttest measure. Staff-reported measures were determined to have changed after 
participation in student listening circles if computed difference scores were statistically different from zero. 

Source: Authors' analysis of primary data collected for the study through surveys. 


of staff who agreed or strongly agreed that students can be counted on (trust in students) 
increased from 79 percent before to 86 percent one week after the student listening circles. 

Twelve weeks after participating in student listening circles, staff reported a higher average 
perception that students have opportunities for meaningful participation at school and 
competency in school improvement efforts and higher trust in students (table 6). Of the 
five measures assessed, staff perceptions of student competency for school improvement 
increased the most after the student listening circle. 

No significant changes were seen in staff perceptions of student-staff relationships and 
in staff beliefs in their own ability to improve the school after the student listening 
circle. There were no statistically significant differences in staff perceptions of the quality 
of student-staff relationships or of staff’s reported beliefs regarding their own ability to 
improve the school 1 week or 12 weeks after participation in student listening circles com- 
pared with their perceptions before participation (see tables 5 and 6). 


Of the five 
measures assessed 
in the descriptive 
analysis, staff 
perceptions 
of student 
competency 
for school 
improvement 
increased the most 
after the student 
listening circle 


Study schools implemented action plans as intended 


During the action-planning phase of the student listening circle the most common 
actions chosen were to improve the quality of relationships between peers and to 
upgrade school facilities. During the action-planning phase of the student listening 
circles students and staff worked together to identify a few achievable actions intended 
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Table 6. Results of the staff surveys 1 week before and 12 weeks after 
participation in student listening circle, 2015 



Means 

Difference between 
pretest and follow-up 


Staff outcome measure 

Pretest 

(standard 

deviation) 

Follow-up 

(standard 

deviation) 

Difference 3 

(standard 

error) 

p-value 

Standardized 

difference 

Staff 

sample 

size 

Belief that students have 

opportunities for meaningful 
participation at school 

2.71 

2.84 

0.13* 

.05 

0.22 

64 

(0.59) 

(0.74) 

(0.06) 

Student-staff relationships 

3.16 

3.24 

0.09 

.09 

0.20 

64 

(0.43) 

(0.42) 

(0.05) 

Recognition of student 
competency to improve the 

school 

2.65 

2.90 

0.26* 

.00 

0.53 

64 

(0.49) 

(0.47) 

(0.06) 

Trust in students 

1.24 

1.48 

0.24* 

.01 

0.35 

64 

(0.67) 

(0.43) 

(0.08) 

Staff ability to improve the 

3.44 

3.49 

0.05 

.26 

0.13 

64 

school 

(0.42) 

(0.45) 

(0.05) 


* Statistically different from zero at the p < .05 level based on a two-tailed test. 

a. Differences shown in this column may vary from differences calculated from data in the table because of 
rounding. 

Note: Higher numbers indicate more positive perceptions of school climate and competencies (see table 2 for 
descriptions of the staff outcome measures). P-values are based on a regression model that accounted for 
differences across schools and the clustering of observations within schools. To assess changes on staff-re- 
ported outcomes between pretest and follow-up assessment, difference scores were computed by subtracting 
the pretest survey measure from the follow-up survey measure. Staff-reported measures were determined 
to have changed 12 weeks after participation in student listening circles if computed difference scores were 
statistically different from zero. 

Source: Authors' analysis of primary data collected for the study through surveys. 


to build on the momentum of the student listening circle by implementing some school 
improvement activities quickly. Seven of the nine schools selected actions that targeted 
enhancing peer relationships, such as providing time for students to learn about each other 
(for example, a mix-it-up day, when students are asked to sit with someone new during 
lunchtime) or to engage in shared activities. Efforts to improve school facilities (such as 
painting trash cans and upgrading the cafeteria) were other frequently identified common 
actions. Least commonly identified were efforts to increase parental involvement in school, 
to address discipline issues, or to improve classroom instruction. See table B1 in appendix 
B for detailed goals set and actions taken at each study site. 

Schools that implemented a student listening circle followed through with most of their 
planned actions. During interviews school principals and student listening circle coor- 
dinators reported that almost all of the actions identified in the student listening circle 
were addressed in the days following the student listening circle. Some actions were not 
implemented because they were not feasible, such as one school’s effort to increase the 
space available for students in the cafeteria (see table B1 in appendix B). Schools that 
implemented student listening circles also engaged in other practices that were suggested 
during the planning dialogue stage. 


During interviews 
school principals 
and student 
listening circle 
coordinators 
reported that 
almost all of the 
actions identified 
in the student 
listening circle 
were addressed in 
the days following 
the student 
listening circle 
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Implications of the study findings 


Finding no short-term impacts of participation in student listening circles on students’ per- 
ceptions of school climate or on perceptions of their competencies does not mean that 
student listening circles have no value. Student listening circles are intended to start a 
process that may lead to improvements in the overall school climate, including the per- 
ceptions of students, faculty, and staff about schoolwide practices. Moreover, the theory 
of action underlying student listening circles suggests other pathways through which 
they might affect school climate. For example, students who participated in listening 
circles worked with school staff to identify school practices that could result in school 
climate improvements. In many cases, students helped implement these school practices. 
The extent to which listening circles lead to lasting involvement of students in school 
improvement efforts is unclear. Nevertheless, the current study provides no evidence that 
participation in student listening circles leads to some of the main outcomes envisioned 
for student listening circles — increasing students’ perceptions that they have more deci- 
sionmaking power in school improvement, greater competencies to improve the school, 
and better relationships with staff participants in the student listening circles. Study lim- 
itations, described below, may have restricted the ability of the study to observe positive 
impacts. 


Limitations of the study 


The study has a number of limitations. Foremost, it is limited by its primary focus on 
the outcomes of students who participated in listening circles rather than the outcomes 
and experiences of all students in the school. The student listening circle is intended 
to promote improvements in the overall school environment. A rigorous examination 
of the impacts on school characteristics (such as governance, schoolwide practices, and 
school climate) would require a school-level random assignment study. However, there has 
been too little research on student-voice initiatives to warrant conducting a large-scale, 
school-level random assignment study as an initial investigation into the student listening 
circle process. Instead, the current study used a student-level random assignment design 
to examine one aspect of the theory of action underlying the student listening circle (see 
figure 1) — the short-term impacts on the perceptions of students who participated in the 
intervention. 


Finding no short- 
term impacts of 
participation in 
student listening 
circles on students’ 
perceptions of 
school climate or 
on perceptions of 
their competencies 
does not mean 
that student 
listening circles 
have no value 


Another limitation is the possibility that using student-level rather than a school-level 
random assignment may have spillover effects. Estimated impacts on participants may 
be dampened by spillover effects on students not assigned to a student listening circle. 
Students in the control group were likely to be aware that other students and staff were 
collaborating on school improvement activities and may have learned about student lis- 
tening circle activities from acquaintances in the intervention group, which could have 
altered their perceptions of student-teacher relationships and other indicators of school 
climate. Their perceptions of bonding to school, relationships with school peers and staff, 
and perceptions of how much influence students have in decisionmaking at the school 
may have been indirectly influenced by the implementation of a student listening circle 
in the school. Because participating directly in student listening circles was expected to 
be more beneficial to students than simply knowing about the activities, these potential 
spillover effects were anticipated to be weak. 
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Although it is possible that spillover effects reduced the ability to detect short-term 
impacts on students who participated in student listening circles, ancillary analyses sug- 
gested, for the most part, that student perceptions of school climate and competencies did 
not improve for either intervention or control group students following implementation 
of student listening circles. Spillover effects would be expected if both control and inter- 
vention group students benefited from the student listening circle. The only exception 
was that students in both groups reported that higher percentages of staff participants 
in student listening circles knew students’ names 12 weeks after participation than had 
known them before participation — suggesting that both groups of students became more 
familiar to staff. 4 

The study is also limited in that there was no way to rigorously examine impacts on staff 
participants because of the lack of a control group for staff. Standard implementation of 
student listening circles involves participation by staff members holding certain positions 
at a school (principal and counselors) for which there is only one incumbent. As a result, 
neither the effects of the intervention on the perceptions and outcomes of staff in those 
positions nor the impacts of student listening circles on practices aimed at improving school 
climate can be ascertained without conducting a school-level randomized trial. Conse- 
quently, the descriptive analyses of the staff surveys need to be interpreted with caution, 
and their generalizability is limited. With no staff control group, it is not possible to ascer- 
tain how staff participants’ perceptions would have changed if the student listening circles 
had not taken place. Thus the survey data provide no information about the impacts of 
participation on staff perceptions. Rather, the results simply describe changes in staff partic- 
ipants’ perceptions during the period when the student listening circles occurred. 

Moreover, the survey responses may have been influenced by the fact that participating 
staff were selected by their principals. The responses of these staff may differ from the 
responses of other staff in a school. Survey attrition rates were also high. Of the 110 school 
staff participants who were available to participate, 68 completed the pretest and posttest 
surveys (1 week before and after participation), and 64 completed the pretest and follow-up 
surveys (1 week before and 12 weeks after participation). 

A further limitation of the study is the self-selection of schools and student participants. 
The results may not generalize to schools without an interest in implementing student lis- 
tening circles or to students in schools that are implementing student listening circles who 
do not wish to participate in them. Historically, participation in a student listening circle 
has been voluntary, and although it would be hard to conceive of a situation in which a 
student would be required to participate, the self-selection of student participants into this 
study may have diluted or amplified short-term program impacts. The intervention was 
described to all potential student participants, and students were asked to consider partic- 
ipation in the study only if they were willing to participate in the student listening circle. 
Consequently, students who chose to participate might be expected to be more engaged in 
school from the onset, more invested in school improvement, and more willing to share 
their opinions than nonparticipating students might be. Such self-selection factors may 
have limited increases in students’ perceptions of school climate and competencies as a 
result of participating in student listening circles. Alternatively, students who volunteered 
may have been more receptive to the intervention than nonvolunteers; it is conceivable 
that the student listening circle would have stronger impacts on students who are more 
receptive to it. 


The self-selection 
of schools 
and student 
participants in the 
study means that 
the results may 
not generalize to 
schools without 
an interest in 
implementing 
student listening 
circles or to 
students in 
schools that are 
implementing 
student listening 
circles who 
do not wish to 
participate in them 
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Appendix A. Research design, outcome measures, and analysis methods 


This appendix explains the research design and data, outcome measures, and analysis 
methods that were used to address the primary experimental research question and the 
two descriptive questions. Each section focuses first on the impact study (addressing the 
first research question) and then on the descriptive portion of the study (addressing the 
second and third research questions). 

Research design and sample characteristics 

For the impact study portion of the research a pretest-posttest control group design (Campbell 
& Stanley, 1963) was used to assess student listening circle (SLC) impacts on student out- 
comes (table Al). The intervention consisted of participation in an SLC workshop in spring 
2015, and data were collected through surveys administered to intervention and control groups 
before and after the student listening circles occurred. Posttest surveys were administered one 
week after student listening circles to assess any immediate effects of participation, and fol- 
low-up surveys were administered at 12 weeks to estimate the temporal stability of any effects. 

The School Climate Alliance (SCA) recruited students for the study and conducted all 
data collection activities, with support from Regional Educational Laboratory (REL) West. 
The study took place during the 2014/15 school year over a 14-week period at nine middle 
schools in eight California school districts: Colton Joint Unified School District, Hemet 
Unified School District, Huntington Beach City School District, Poway Unified School 
District, San Diego Unified School District, St. Helena Unified School District, Victor 
Valley Union High School District, and West Contra Costa Unified School District. 

The timing of implementation varied across schools according to scheduling constraints, 
but all nine of the student listening circles were completed no later than March 2015 to 
allow time for collecting follow-up data 12 weeks later and before the end of the school 
year. The study took place in middle schools that had not previously implemented a 
student listening circle so that potential changes could be observed in student and staff 
perceptions and in school policies and practices that accompanied SLC implementation. 
To standardize the intervention for the experimental trial, REL West staff provided tech- 
nical support for the implementation of the student listening circle, student recruitment, 
and data collection activities. Data were shared with REL West to conduct the impact 
analyses. 

Student sample recruitment, randomization, and characteristics. Participating schools 
varied in enrollment size and demographic characteristics (table A2). Enrollment ranged 


Table Al. Student listening circle study design and measurement points 



Pretest 

(1 week prior to 
intervention) 


Posttest 
(1 week after 
intervention) 

Follow-up 
(12 weeks after 
intervention) 



Student listening circle 

✓ 

✓ 

Students 

✓ 

Randomized < " „ 





— Control group 

✓ 

✓ 

Staff 

✓ 

Student listening circle 

✓ 

✓ 


✓ indicates survey measurement point. 
Source: Authors' construction. 
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from 288 to 1,449 students across the nine schools. Four of the schools served lower per- 
centages (averaging 27 percent) of students eligible for the federal school lunch program, (a 
proxy for low-income status), than the state average of 59 percent, and five of the schools 
had higher than average percentages of students eligible for the federal school lunch 
program (averaging 85 percent). Five of the schools served students in grades 6-8, and four 
served students in grades 7-8. 

To help recruit students to participate, the principal of each school designated an SLC 
coordinator (usually a school counselor) to work out the logistics of SLC recruitment and 
implementation. Because the goal of the student listening circle is to hear from and involve 
a wide range of students, SLC coordinators were encouraged to seek students who might 
not normally engage in such an activity. SLC coordinators contacted students, described 
the student listening circle and the study, and asked them whether they wanted to par- 
ticipate. Only students who agreed and who had positive consent from a parent (a signed 
consent form) could participate. 

To ensure that student participants represented equal distributions of students across grade 
and gender, each school, with REL West’s assistance, organized student volunteers into 
groups (or “randomization strata”) that consisted of one group of male students and one 
group of female students for each grade level in the school — totaling six grade/gender groups 
for the schools serving grades 6-8 and four grade/gender groups for the schools serving 
grades 7-8. These groups were then randomly divided into intervention and control groups. 
Within each randomization stratum in each school, approximately one to three students 
were randomly assigned to the intervention group (the students who would participate in 
the SLC workshop), and the remaining students served as the control group (not assigned 


Table A2. Characteristics of schools participating in the student listening circle 

study, 2015 




School characteristic All schools 

Low-income schools 

High-income schools 

Number of schools 

9 

5 

4 

Grade configuration (number of schools) 

6-8 

5 

1 

4 

7-8 

4 

4 

0 

Student enrollment (number) 

846 

825 

863 

Race/ethnicity (percent) 

Asian 

5 

2 

8 

Black 

10 

4 

15 

Hispanic 

52 

69 

31 

White 

16 

11 

49 

English learner status 

English learner students (percent) 

16 

21 

10 

Eligibility for the federal school lunch program 

Eligible (percent) 

59 

85 

27 


Note: For student enrollment, race/ethnicity, English learner students, and eligibility for the federal school 
lunch program the numbers are averages across the set of schools in each school category. Race/ethnicity 
percentages do not sum to 100 because some categories (such as Filipino, two or more races, and none re- 
ported) are omitted. Low-income schools are those with 50 percent or more of students eligible for the federal 
school lunch program, and high-income schools are those with less than 50 percent of students eligible. 

Source: California Department of Education, 2015. 


A-2 








to participate in the student listening circle). Because more girls than boys were in the ran- 
domization pool, boys had a higher probability of being assigned to the intervention group. 5 

Because effective implementation of the student listening circle requires a minimum of 
eight students, an additional two control students per stratum were randomly selected to 
serve as a replacement pool for SLC participants if more than two students assigned to the 
intervention group did not participate. Students in the replacement pool were randomly 
selected within each applicable stratum to participate in the intervention on the day of 
the student listening circle until a total of eight SLC participants were available. 6 If eight 
or more intervention group students were available on the day of the student listening 
circle, replacement students did not participate in the student listening circle. To provide 
unbiased comparisons between the intervention and control groups, intervention group 
“no-shows” and replacement group students were kept in their original random assignment 
groups for the analyses to account for nonrandom attrition or students crossing conditions. 7 

Of the 256 students who were recruited and had parental consent to participate in the 
study, 90 were randomly assigned to the intervention group and 166 were assigned to the 
control group (figure Al). Of the 90 students assigned to the intervention group, 13 were 
absent or unavailable to participate in the student listening circle on the day it took place. 


Figure Al. Student sample numbers, allocation, and attrition, 2015 
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Source: Authors’ analysis of primary data collected for the study through participation rosters and surveys. 
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Consequently, four students from the control group replacement pool participated in the 
student listening circle. 

All students who provided posttest or follow-up survey data were included in the impact 
analysis samples. Student posttest surveys were collected from 234 students: 84 students 
(93 percent) in the intervention group and 150 students (90 percent) in the control group. 
Follow-up survey data were collected from 237 students: 86 intervention group students 
(96 percent) and 151 control group students (91 percent). 

Student sample characteristics and baseline equivalence. Because four of the nine par- 
ticipating schools do not serve grade 6 students, fewer grade 6 students participated in the 
study than grade 7 and grade 8 students (table A3). Twenty percent of study participants 
were in grade 6, 42 percent in grade 7, and 38 percent in grade 8 — with similar percentages 
across intervention and control groups. Sixty percent of study participants were female and 
40 percent were male. Because roughly equal numbers of female and male students were 
randomly assigned to the student listening circle, the intervention group had a smaller 
percentage of female students (54 percent) than the control group had (64 percent). This 
difference is not statistically significant (see note 5). Forty-seven percent of student study 
participants indicated that they were Hispanic, 21 percent were White, 16 percent were of 
mixed race (two or more races), 8 percent were African American, and 8 percent “other.” 
Differences in reported racial/ethnic group status across the intervention and control 
group were not statistically significant. 


Intervention and control group students had similar pretest averages on the measures of 
school climate perceptions and competencies (table A4). There were no statistically signif- 
icant differences across groups on these measures. 


Table A3. Characteristics of student participants in the student listening circles, 2015 

Characteristic 

Total sample 
(percent) 

Intervention 

group 

(percent) 

Control group 
(percent) 

p-value 

Student 
sample size a 

School grade 

6 

20.1 

17.9 

21.3 

.78 

234 

7 

41.9 

41.7 

42.0 



8 

38.0 

40.5 

36.7 



Gender 

Female 

60.3 

53.6 

64.0 

.13 

234 

Male 

39.7 

46.4 

36.0 



Race/ethnicity 

African American 

8.1 

8.0 

8.0 

.69 

234 

Flispanic 

47.0 

47.6 

46.7 



White 

21.4 

20.2 

22.0 



Mixed 

15.8 

13.1 

17.3 



Other 

7.7 

10.7 

6.0 




a. Sample size is the number of students in the final analytic sample that was used for the posttest impact 
analysis. School grade and gender data were collected from roster data to conduct the random assignment. 
There were no missing race/ethnicity data because this characteristic was assessed on the pretest, posttest, 
and follow-up surveys. 

Note: The p-values, based on Fisher’s exact test computations, represent the probability that the differences 
in distributions across the intervention and control groups are due to chance. 

Source: Authors' analysis of primary data collected for the study. 
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Table A4. Pre-intervention sample characteristics for the posttest analytic sample 
for student listening circle participants, 2015 



Means 

Difference between 
intervention and control groups 


Student 

outcome measure 

Intervention 

group 

(standard 

deviation) 

Control 

group 

(standard 

deviation) 

Difference a 
(standard 
error) 

p-value 

Standardized 

difference 

Student 

sample 

size b 

School climate perceptions 

Student participation in 

2.16 

2.20 

-0.03 

.65 

-0.05 

227 

decisionmaking 

(0.66) 

(0.66) 

(0.08) 

Student-staff relationships 

SLC staff 0 knows 

0.41 

0.40 

0.01 

.71 

0.04 

229 

student’s name 

(0.28) 

(0.25) 

(0.03) 

Student is comfortable 

2.44 

2.52 

-0.08 

.58 

-0.08 

222 

talking to SLC staff 0 

(1.11) 

(1.07) 

(0.15) 

Caring relationships 

3.17 

3.22 

-0.05 

.52 

-0.09 

229 

(0.64) 

(0.62) 

(0.08) 

School bonding 

2.85 

2.95 

-0.11 

.15 

-0.20 

227 

(0.65) 

(0.54) 

(0.08) 

Peer support 

2.54 

2.59 

-0.04 

.63 

-0.07 

229 

(0.68) 

(0.64) 

(0.09) 

Competencies 

Competency in school 

2.82 

2.88 

-0.06 

.36 

-0.13 

227 

improvement efforts 

(0.52) 

(0.48) 

(0.36) 

Academic self-efficacy 

3.72 

3.74 

-0.02 

.86 

-0.03 

226 

(0.86) 

(0.80) 

(0.12) 


a. Difference may not be equal to calculated difference between the intervention and control group values due 
to rounding error. 

b. The final analytic sample that was used for the posttest impact analysis and with non-missing pretest data. 

c. SLC staff are staff members who participated in the student listening circle. 

Notes: Regression models that accounted for study design characteristics (randomization strata and school) 
were estimated to obtain p-values for intervention/control group differences. 

Source: Authors’ analysis of primary data collected for the study. 


Staff sample. Across the nine schools, 148 adults attended the student listening circle, of 
which 110 were school staff eligible to participate in the study. The remaining adult partici- 
pants were district staff (15), parents (8), or other adults (15, which included staff from other 
schools, members of the community, and others). Fifty-seven percent of school staff SLC 
participants were teachers (table A5). Seven of the nine student listening circles had a school 
principal or associate principal participate, and six of the student listening circles had at least 
one school counselor participate. (Two counselors participated in the SLC in four schools.) 

Paper-and-pencil pretest staff surveys were administered in all nine schools. Posttest and 
follow-up surveys were administered online. Posttest and follow-up surveys were not admin- 
istered in two of the nine schools because of logistical constraints related to email spam 
filtering. Pretest survey data were collected from 87 staff study participants (79 percent of 
the school staff who participated in the student listening circles), posttest survey data from 
69 participants (63 percent of the staff), and follow-up survey data from 65 participants 
(59 percent of the staff; figure A2). 






Table A5. School roles of staff participants in the student listening circles, 2015 


1 Role 

Number 

Percent | 

Teacher 

63 

57.3 

Instructional aide 

11 

10.0 

Counselor 

10 

9.1 

Classified staff 

9 

8.2 

Certificated staff 

7 

6.4 

Principal 

7 

6.4 

Total 

110 

100.0 

Source: Authors' analysis of primary data collected for the study through participation rosters. 


Figure A2. Adult study participant numbers and attrition, 2015 
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SLC participants are those who participated in a student listening circle. 

Source: Authors’ analysis of primary data collected for the study through participation rosters and surveys. 


Outcome measures 

The study outcomes were measured by comparing results from student and staff surveys 
(see appendix D) that SCA districts administered up to 1 week prior to assignment of 
students to intervention and control groups (pretest) against the results of surveys adminis- 
tered 1 week after SLC implementation (posttest), and 12 weeks after SLC implementation 
(follow-up). Also, SCA coordinators documented the actions that were suggested during 
the student listening circles and any schooHevel actions, policies, and practices that result- 
ed from the SLC implementation as the school year unfolded. 
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Student measures. The study examined the student listening circle’s effects on two 
outcome domains: students’ perceptions of school climate and of their own competencies. 
In the following discussion of outcomes that were measured, the first five were used to 
assess the school climate perceptions domain and the last two were used to assess the com- 
petencies domain. Within each domain, the list of outcomes is arranged with the outcomes 
expected to be most affected by the intervention described first. The baseline student 
survey (conducted prior to students’ assignment into intervention and control groups) also 
collected information on student demographic characteristics. Outcome indices for each 
subdomain were constructed by averaging the items that comprise the subdomain. 

Student participation in decisionmaking. Students’ perception of their input into decision- 
making was assessed by means of a scale developed by Brand, Felner, Shim, Seitsinger, and 
Dumas (2003) and adapted for this study. Students rated the statement “Students in this 
school have a say in how things work” by selecting one of four response categories ranging 
from 1 (strongly disagree) to 4 (strongly agree). 

Student-staff relationships. Two instruments assessed students’ perceptions of staff rela- 
tionships with students. First, students were asked to rate student ties with specific staff 
members using a social network questionnaire. Using a grid system that listed up to 44 
staff members, including the SLC staff participants, 8 students were asked to indicate 
whether each staff member knew the student’s name and whether the student felt comfort- 
able talking to the staff member about things that were bothering the student. Although 
the survey questions asked about all or most staff members in each school, only the items 
referencing staff SLC participants were used to create outcome measures. Students’ ratings 
were used to compute student-level averages of relationship ties with staff SLC participants 
only. Second, a six-item scale from the California School Climate, Health, and Learn- 
ing Survey (Cal-SCHLS) system student survey was used to assess students’ perceptions 
of student-staff relationships (Hanson & Voight, 2014). The items asked students to rate 
statements about caring relationships with adults (“At my school, there is a teacher or some 
other adult who really cares about me”) and supportive, high-expectations messages from 
adults at school (“At school, there is a teacher or some other adult who tells me when I do a 
good job”). Response options range from 1 (not at all true) to 4 (very much true). 

School bonding. Students’ perceptions of school bonding were assessed using survey items 
from the identification/participation in school subscale (Ye & Wallace, 2014) from the 
Psychological Sense of School Membership scale (Goodenow & Grady, 1993) that were 
adapted for this study. This subscale consists of six items (for example, “I feel like a real 
part of this school”) that measure school bonding, with response categories ranging from 1 
(not at all) to 5 (completely true). The response categories were modified to assess strength 
of agreement (1 = strongly disagree, 4 = strongly agree) to be consistent with other mea- 
sures used in the study surveys. 

Peer support. Williams and Guerra’s (2007) six-item scale was used to assess students’ per- 
ceptions of peer support. Adapted from the Generalized Perceptions of Peers scale (Salmi- 
valli, Ojanen, Haanpaa, & Peets, 2005), the Williams-Guerra scale captures both positive 
and negative qualities of peers as a source of social support (for example, “Students my age 
really care about what happens to me/can be trusted a lot/only think about themselves”). 
The original response options ranging from 1 (no, not at all) to 4 (yes, completely) were 
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altered to “not at all true,” “a little true,” “pretty much true,” and “very much true” for 
consistency with other items on the study survey. 

Competencies for improving the school. Students’ beliefs about their competencies to 
improve the school were measured using survey items adapted from the policy control sub- 
scale of the Sociopolitical Control Scale for Youth (Peterson, Peterson, Agre, Christens, & 
Morton, 2011). This nine-item scale measures students’ perceived ability to effect change 
in their community or school (Christens, Peterson, Reid, & Garcia-Reid, 2012). Example 
items include: “I feel like I have a pretty good understanding of the important issues which 
confront my community or school” and “Most community or school leaders would listen 
to me.” The items were altered to replace “community or school” with “school” so that the 
items assessed student empowerment exclusively in the school setting. Response categories 
ranged from 1 (strongly disagree) to 4 (strongly agree). 9 

Academic self efficacy. Students’ perceptions of their academic self-efficacy were assessed 
with survey items from an eight-item subscale developed by Muris (2001, 2002). The items 
assessed student perceptions of academic efficacy irrespective of academic content (for 
example, “How well can you get teachers to help you when you get stuck on homework?”). 
Students responded to five-category Likert items ranging from 1 (not at all) to 5 (very well). 

Staff measures. Surveys of the staff SLC participants were conducted at the same times 
as the student surveys: 1 week prior to SLC implementation (pretest), 1 week after SLC 
implementation (posttest), and 12 weeks after SLC implementation (follow-up). This 
section describes the outcomes measured by the staff surveys. The pretest surveys asked 
for the respondent’s role in the school (teacher, counselor, principal) in addition to the 
measures described below. Indices were constructed by averaging the items that comprised 
the subdomain. 

Student-staff relationships. Staff perceptions of student-staff relationships were assessed 
using items adapted from Brand, Felner, Seitsinger, Burns, and Bolton (2008). These items 
ask teachers about the frequency of interactions with students involving personal issues 
(for example, “Students. . .share their concerns with me/express their feelings to me/talk 
with me about their homes and families”). The response categories for these items range 
from 1 (strongly disagree) to 4 (strongly agree). 

Student opportunities for meaningful participation at school. Two sets of items were used to 
assess staff perceptions of students’ opportunities for meaningful participation at school: 
an item from the Cal-SCHLS system staff survey (“This school encourages opportunities 
for students to decide things like class activities or rules”) and three modified items from 
the Alaska School Climate and Connectedness Survey (ASCCS) developed by Ameri- 
can Institutes for Research (2013) (for example, “Students are involved in helping to solve 
school problems”). Although both sets of items have been used independently, and both 
demonstrate adequate reliability (alpha = 0.81-0.83), Hanson and Voight’s (2014) analysis 
suggests that the Cal-SCHLS scale did not contain sufficient items at that time to assess a 
dimension of staff perceptions of student involvement distinct from other school climate 
factors. To ensure sufficient construct coverage, the Cal-SCHLS and ASCCS items were 
combined to form a seven-item scale, using the Cal-SCHLS response categories (1 = 
strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree). 10 
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Recognition of student competencies for improving the school. Because the study team was 
unable to locate validated measures of teacher perceptions of student competency for school 
improvement, items were developed to assess this outcome (for example, “Students have 
unique perspectives about barriers to learning”). Several items were adapted from Caprara, 
Barbaranelli, Borgogni, and Steca’s (2003) teacher efficacy beliefs scale and Van Maele and 
Van Houtte’s (2011) teacher trust in students scale. The CahSCHLS response categories 
were used for these items (1 = strongly disagree, 2 = disagree, 3 = agree, 4 = strongly agree). 

Trust in students. A measure developed by Hoy and Tschannen-Moran (1999) and adapted 
by Van Maele and Van Houtte (2011) was used to assess staff trust in students. The measure 
includes 10 strength-of-agreement items measured on a four-point scale. Examples of items 
include “Students in this school are reliable” and “Students in this school can be counted 


Staff efficacy to improve the school. Items from the efficacy subscale from Short and Rine- 
hart’s (1992) School Participant Empowerment Scale were used to assess staff perceptions 
of their commitment to student development (for example, “I believe that I am empower- 
ing students,” “I perceive that I am making a difference”). The measures include six items, 
each with four response categories measuring strength of agreement (1 = strongly disagree, 
2 = disagree, 3 = agree, 4 = strongly agree). 

Documentation of practices implemented as a result of student listening circles. As 
described earlier, the student listening circle concludes with an action-planning phase, in 
which students and staff identify two to four concrete, achievable, time-limited actions to 
improve school climate. The actions were documented as part of the SLC process. In addi- 
tion, structured interviews were conducted with administrators in late spring after SLC 
implementation to assess their perceptions of these practices. 

Analytic methods 

SCA school districts shared all collected data with REL West, and the REL West study 
team conducted all data analyses because SCA districts do not have the resources or 
capacity to conduct such analyses. 

Impact analyses of student reported measures. The analysis of short-term SLC impacts 
on student outcomes (research question 1) depends on the random assignment research 
design as its primary source of inference. The primary hypothesis-testing analyses involved 
fitting single-level regression models. Covariates included intervention group, baseline 
measures of outcome variables, other observed covariates (gender and grade), and fixed 
effects for school and randomization strata. The model takes the following form: 

DV ,j = a o + Pi PreDV + [2 2 SLC + IjSjT + I^Stram + I ^School + t 

1=2 

where subscripts i and j denote student and school, respectively; DV represents the student 
outcome, measured either 1 week or 12 weeks after SLC implementation; PreDV represents 
the pretest measure of the outcome; SLC is a dichotomous variable indicating student 
random assignment to the intervention group; I is a vector (or control variable) for stu- 
dents measured prior to exposure to the intervention; Strata represents dichotomous vari- 
ables for each randomization stratum and accounts for unequal probabilities of assignment 
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to the intervention group; School represents dichotomous variables for each school; and £.. 
is an error term for individual students. The j dichotomous variables account for variation 
in the mean value of the dependent variable across participating schools. The intervention 
effect is represented by (2 2 , which captures intervention/control differences in the outcome 
variable. Impacts at 1 week and 12 weeks after SLC implementation were estimated in 
separate models. 

Treatment of missing data. The missing-indicator method was used to account for missing 
values on the covariates in the student impact analysis models (White & Thompson, 
2005). In the missing-indicator method all observations with missing values on covariates 
are retained in the analysis. Indicator variables were created for missing values on each 
variable (0 = observed, 1 = missing), and missing values on the covariates were coded to 
a constant. Both the recoded covariates and the missing value indicator variables were 
included in the impact analysis model. Observations with missing values on outcome 
variables were excluded from the impact analyses. Deletion of observations with missing 
outcome variables has been shown to result in accurate impact estimates and standard 
errors when outcomes are missing at random, conditional on the covariates (Allison, 
2002; von Hippel, 2007). Staff with missing data on the applicable pretest or the posttest 
measure were excluded from the analyses examining change between pretest and posttest. 
Those with missing data on the pretest survey measure and on the follow-up measure were 
eliminated from the analysis examining follow-up data. 

Multiple hypothesis testing. Two outcome domains were delineated for research ques- 
tion 1: school climate and competencies. Within these two outcome domains, multiple 
comparison procedures were used to adjust for the inflation of type 1 errors due to multi- 
ple hypothesis tests. Adjustments were applied to adjust for the six statistical tests for the 
school climate outcomes and for the two statistical tests for the competencies outcomes. 
No across-domain adjustments were made. Benjamini and Hochberg’s (1995) stepwise mul- 
tiple hypothesis testing procedure was used to adjust for multiple statistical tests. This pro- 
cedure involves ordering p-values obtained for each outcome variable within each domain 
from largest to smallest and multiplying each unadjusted p-value by N/(N - j + 1), where N 
is the number of outcome variables within a domain and j represents the order of the test. 
The procedure involves rejecting all null hypotheses in which the adjusted p-value is less 
than .05. Because research question 2 is descriptive, no multiple comparison adjustments 
were applied when analyzing the staff survey data. 

Treatment of replacements. As described previously, if fewer than eight students assigned 
to the intervention group were available when the student listening circle took place, 
intervention “no shows” were replaced with randomly selected control group students. 
Thirteen students assigned to the intervention group did not participate in the student 
listening circle due to absence or unwillingness to stay after school, and four randomly 
selected control group students participated in the student listening circle in their place. 
These 17 students were kept in their original random assignment groups in the analysis. 

The study team conducted a sensitivity analysis to see if students who participated in the 
student listening circle (regardless of their random assignment status) exhibited higher 
regression-adjusted averages on posttest and follow-up outcomes than did students who did 
not participate in the student listening circle. The regression model took the same form 
as the impact analysis model described earlier, except an indicator for SLC participation 
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was used instead of the random assignment indicator. The point-estimates and statistical 
significance levels from this descriptive model did not differ from the impact analysis 
results. 

Descriptive analysis of student survey responses. To assess the possibility that SLC 
implementation may have influenced students’ perceptions of school climate and compe- 
tencies in both the intervention and control groups, descriptive analyses were conducted to 
examine the extent to which perceptions changed in both groups. Changes on student-re- 
ported outcomes between pretest and posttest and changes between pretest and follow-up 
assessments were examined by computing difference scores by subtracting the pretest 
survey measure from the appropriate postintervention measure (Y - Y , Y f „ - Y ). 

Difference scores were examined separately for the complete sample, the control group, 
and the intervention group. Student-reported measures were determined to have changed 
after participation in student listening circles if computed difference scores were statis- 
tically different from zero. (The point estimates using this procedure are the same that 
would be obtained by calculating paired t-tests.) To obtain standard errors of difference 
scores, analyses accounted for the dependence among students within schools by using the 
Huber-White sandwich estimator of variance (Huber, 1967; White, 1980). 

Descriptive analysis of staff survey responses. Staff survey data were used to examine 
staff SLC participants’ perceptions of school supports and perceptions of student compe- 
tencies. Analyses similar to those described above for students were conducted to assess 
changes on staff-reported outcomes. Because there is no comparison group, these results 
simply describe the changes in the adult SLC participants’ survey responses. The results 
cannot be used to make inferences about SLC impacts on staff. 

Implementation and follow-up analyses. The study team facilitated the student listening 
circle at each school to help ensure that the intervention was delivered consistently across 
sites. Facilitators took notes about each SLC component session, noting both typical and 
special circumstances. A critical outcome of each SLC planning dialogue is a list of short- 
and long-term action items. Critical themes addressed during the planning dialogue and 
recommended follow-up actions were recorded on a planning dialogue summary template. 
Information recorded on the template was summarized and used as the basis for follow-up 
structured interviews with administrators to assess perceptions of practices implemented as 
a result of the student listening circle. 

Template data and administrator interview data were examined to identify the extent 
to which SLC actions were implemented, potential barriers to follow-through with the 
actions, and possible other actions perceived to have been taken as a result of the student 
listening circles. These results are purely descriptive and cannot be used to make infer- 
ences about impacts of the student listening circle on school practices. The actions sug- 
gested during student listening circles and their dispositions are listed in appendix B. 
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Appendix B. Student listening circle goals set and actions taken 


The school-improvement actions proposed by student listening circles and their disposition 
are shown in table B1 by school. The action plans were recorded by the study team during 
the student listening circles. Follow-up interviews with school administrators determined 
whether and how each proposed action was carried out. Most actions were carried out at 
least in part. 


Table Bl. Student listening circle goals set and actions taken, 2015 


School Short-term action plan goal Subsequent activities 


A 1. Develop parent/ student survey assessing time needed for 

homework (instruction). 

2. Remove dirt pile from physical education/running area on 
campus (facilities, safety). 


1. Developed surveys for students, parents, and staff 
(instruction). Collaborated with high school to get more 
information on homework (instruction). 

2. Leveled jogging path and removed dirt pile (facilities, 
safety). 

3. Held assembly about behavioral expectations discussed in 
student listening circle (discipline). 

4. Investigated restroom facility improvement (facilities). 


B 1. Conduct needs assessment regarding student clubs (peer 

relationships). 

2. Share SLC results with all staff (dissemination, 
relationships). 

Postponed implementation until fall 2015 (peer 
relationships). 

5. Implemented lunch-table activities to encourage student/ 
staff interaction (relationships). 

6. Planned periodic student assemblies/spirit days/rallies to 
promote school connectedness and relationships between 
students and staff (relationships). Implemented one 
assembly. 


1. Developed survey, sent to staff to identify club sponsors. 

2. Shared results at staff meeting. 

3. Formed school climate committee composed of staff and 
students. 

4. Planned once-a-week icebreaker activities for students. 


C 1. Have student-led morning announcements (meaningful 

participation). 

2. Strategically post cafeteria menu to eliminate delays in 
lunch line (facilities). 

3. Add more space/tables in lunch area (facilities). 


1. Did not implement student-led morning announcements. 

2. Posted menu daily on a sign outside the cafeteria so 
students have time to contemplate food choices. 

3. Did not implement lunch area expansion. 

4. Switched to having food entirely prepared on-site, no more 
prepackaged food (facilities). 

5. Shared SLC results at an all-staff meeting (relationships). 

6. Faculty was encouraged to and began using more 
cooperative, hands-on, or project-based learning 
(instruction). 

7. Family Night became more project-based because of SLC 
results (instruction, parent involvement). 

8. Re-implemented Science Fair (instruction, project-based 
learning). 

9. Implemented a "Dancing Through the Decades” school 
activity (relationships). 


(continued) 
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Table Bl. Student listening circle goals set and actions taken, 2015 (continued) 

1 School 

Short-term action plan goal 

Subsequent activities 1 

D 

1. Create an Instagram account for school to reduce cyber 
bullying (safety). 

2. Purchase new sports equipment (facilities). 

3. Share SLC ideas with the Associated Student Body for 
more group collaboration (dissemination, relationships). 

4. Implement surveys/suggestion box/ polls to gain more 
student input (meaningful participation). 

1. Did not implement Instagram account. 

2. Purchased some new sports equipment. 

3. Shared SLC highlights/findings with Associated Student 

Body and at all staff meeting. 

4. Did not implement. 

E 

1. Implement a mix-it-up day at lunch to increase cross-grade 
interaction (peer relationships). 

2. Implement lunchtime activities/clubs (peer relationships). 

3. Develop survey to learn of student/ staff interests. 

4. Put forth teacher-student consequences proposals with 
learning community agreements (discipline). 

5. Acquire board games through family donations to help 
integrate grade levels (peer relationships). 

1. Student council planned and implemented at least one 
mix-it-up day. 

2. Implemented Friday music at lunch; also attempted to have 
chess club with visiting adults at lunch, but moved to after 
school to allow for more time. 

3. Did not implement survey. 

4. Did not implement teacher-student consequences proposal. 

5. Acquired board games and made them available in the 
library. 

F 

1. Open up rooms to spend time at lunch (peer relationships). 

2. Create an anti-bullying club (safety, peer relationships). 

3. Create "Trash Can Competition" to promote a cleaner 
environment (facilities). 

1. Many teachers offered to open up classrooms during lunch; 
but not implemented because of teacher contract issues. 

2. Created an anti-bullying club, which meets 1-2 times 
per week, attended by 20-30 students per meeting. 
Implemented various activities (Give-a-Compliment 

Challenge Day, Warm and Fuzzy Day) and provided 

coordinator with testimonials. 

3. Principal offered to pursue purchasing more trash cans 
and provide one for each club, but interest in this project 
subsided. 

G 

1. Implement Mix-it-up Wednesdays (peer relationships). 

1. Implemented Mix-it-up Wednesdays, which has included 
cross-grade groups with ice-breaker activity (two truths and 
a lie) and cooperative task (cup stacking with string). 

H 

1. Recognize positive attitudes by handing out tokens (peer 
relationships). 

1. Select SLC participants with Positive Behavioral 

Interventions and Supports team to plan roll-out of token 
system. 

2. Continued meeting with staff and students to implement 
token system in 2015/16. 

3. Shared themes and topics of Student Listening Circle at 
staff meeting (dissemination, staff relationships). 

1 

1. Implement lunchtime activities for students (peer 
relationships). 

2. Address the overall theme of relationship-building (staff and 
student relationships). 

1. Developed a plan to get students more engaged during 

lunch. 

2. Established student voice planning committee to plan 
activities for 2015/16; administrative team met with 
classified staff to offer ways to improve the front-office 

climate. 

3. Trained teachers on best practices in working with students 
who face trauma; gave classroom presentations on stress 
management as a way to help with coping skills (overall 
climate). 

Note: Terms in parentheses indicate school climate theme addressed. 

Source: Authors' summaries of action-planning items and interview data collected for the study. 
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Appendix C. Ancillary analyses of student surveys 


This appendix shows student survey results for surveys administered 1 week before the 
student listening circle (SLC), 1 week after it (table Cl), and 12 weeks after it (table C2), 
for the total sample and by intervention status. The results indicate that average student 
perceptions of school climate and competencies prior to the student listening circle did not 
statistically differ one week later for either the control group or the intervention group. 
At 12 weeks after the student listening circle, students in both intervention and control 
groups reported that higher percentages of staff who attended the student listening circle 
knew their names than had known them prior to the student listening circle. Students 
in the intervention group, but not those in the control group, reported higher average 
levels of participation in decisionmaking and greater average levels of comfort talking 
with staff participants in the student listening circles at the 12'week follow-up than in the 
pretest. Although students in the intervention group experienced a statistically significant 
increase in participation in decisionmaking, the increase was not statistically significant- 
ly greater than that experienced by students in the control group. Thus perceptions of 
decisionmaking opportunities did not increase more among students in the intervention 
group than among students in the control group. The statistically significant increase in 
intervention students’ comfort level when talking with staff participants in the student 
listening circles is consistent with the experimental results in table 4 in the main text (see 
endnote 2). Both groups reported lower average levels of student competency for school 
improvement and academic self-efficacy at the 12-week follow-up compared with their 
reported competencies prior to the student listening circle. 


With the exception of perceptions that staff SLC participants knew students’ names, the 
results do not suggest that all participating students experienced gains with respect to per- 
ceptions of school climate and competencies after the student listening circle. 


Table Cl. Results of the student surveys conducted 1 week before and 1 week 
after participation in a student listening circle, 2015 





Difference between 



Means 

pretest and posttest 



Pretest 

Posttest 

Difference 3 



Student 


(standard 

(standard 

(standard 


Standardized 

sample 

Student outcome measure 

deviation) 

deviation) 

error) 

p-value 

difference 

size 

School climate perceptions 

Student participation in decisionmaking 

All 

2.19 

2.23 

0.04 

.45 

0.06 

227 

(0.66) 

(0.65) 

(0.03) 

Control group 

2.18 

2.23 

0.05 

.45 

0.07 

144 

(0.66) 

(0.65) 

(0.04) 

Intervention group 

2.19 

2.22 

0.03 

.69 

0.04 

83 

(0.66) 

(0.66) 

(0.06) 

Student-staff relationships 

Knows name (SLC staff 0 ) 

All 

0.41 

0.38 

-0.02 

.40 

-0.08 

229 

(0.26) 

(0.22) 

(0.01) 

Control group 

0.40 

0.37 

-0.02 

.39 

-0.08 

145 

(0.25) 

(0.22) 

(0.02) 

Intervention group 

0.42 

0.40 

-0.02 

.49 

-0.07 

84 

(0.28) 

(0.23) 

(0.02) 

(continued) 
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Table Cl. Results of the student surveys conducted 1 week before and 1 week 
after participation in a student listening circle, 2015 (continued) 





Difference between 



Means 

pretest and posttest 



Pretest 

Posttest 

Difference 3 



Student 


(standard 

(standard 

(standard 


Standardized 

sample 

Student outcome measure 

deviation) 

deviation) 

error) 

p-value 

difference 

size 

Comfort talking (SLC staff b ) 

All 

2.49 

2.59 

0.10 

.22 

0.09 

222 

(1.08) 

(1.06) 

(0.05) 

Control group 

2.51 

2.56 

0.06 

.53 

0.05 

143 

(1.07) 

(1.05) 

(0.06) 

Intervention group 

2.47 

2.64 

0.17 

.23 

0.15 

79 

(1.11) 

(1.09) 

(0.10) 

Caring relationships 

All 

3.20 

3.22 

0.02 

.68 

0.04 

229 

(0.63) 

(0.66) 

(0.03) 

Control group 

3.19 

3.21 

0.02 

.76 

0.03 

145 

(0.62) 

(0.63) 

(0.04) 

Intervention group 

3.21 

3.24 

0.03 

.61 

0.05 

84 

(0.64) 

(0.71) 

(0.05) 

School bonding 

All 

2.91 

2.87 

-0.04 

.13 

-0.08 

227 

(0.58) 

(0.60) 

(0.02) 

Control group 

2.93 

2.88 

-0.05 

.16 

-0.10 

144 

(0.54) 

(0.57) 

(0.03) 

Intervention group 

2.89 

2.86 

-0.03 

.37 

-0.05 

83 

(0.65) 

(0.64) 

(0.04) 

Peer support 

All 

2.57 

2.54 

-0.03 

.50 

-0.05 

229 

(0.66) 

(0.63) 

(0.03) 

Control group 

2.60 

2.58 

-0.02 

.65 

-0.03 

145 

(0.64) 

(0.61) 

(0.04) 

Intervention group 

2.52 

2.47 

-0.05 

.43 

-0.08 

84 

(0.68) 

(0.66) 

(0.06) 

Competencies 

School improvement efficacy 

All 

2.86 

2.80 

-0.06 

.09 

-0.11 

227 

(0.50) 

(0.57) 

(0.03) 

Control group 

2.87 

2.79 

-0.09 

.14 

-0.18 

144 

(0.48) 

(0.57) 

(0.04) 

Intervention group 

2.83 

2.83 

0.00 

.90 

-0.01 

83 

(0.52) 

(0.56) 

(0.04) 

Academic seif-efficacy 

All 

3.73 

3.68 

-0.05 

.06 

-0.06 

226 

(0.82) 

(0.80) 

(0.03) 

Control group 

3.73 

3.67 

-0.06 

.06 

-0.07 

142 

(0.80) 

(0.80) 

(0.04) 

Intervention group 

3.74 

3.70 

-0.05 

.38 

-0.05 

84 

(0.86) 

(0.81) 

(0.05) 


a. Differences shown in this column may vary from differences calculated from data in the table because of 
rounding. 

b. SLC staff are staff members who participated in the student listening circle. 

Note: Higher numbers indicate more positive perceptions of school climate and competencies (see table 1 in 
the main text for descriptions of the student outcome measures). P-values are based on a regression model 
that accounted for clustering of observations within schools. No results were statistically significant. 

Source: Authors' analysis of primary data collected for the study through surveys. 
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Table C2. Results of the student surveys conducted 1 week before and 12 weeks 
after participation in a student listening circle, 2015 





Difference between 



Means 

pretest and follow-up 



Pretest 

Posttest 

Difference 3 



Student 

Student outcome measure 

(standard 

deviation) 

(standard 

deviation) 

(standard 

error) 

p-value 

Standardized 

difference 

sample 

size 

School climate perceptions 

Student participation in decisionmaking 

All 

2.17 

2.20 

0.03 

.53 

0.05 

229 

(0.65) 

(0.65) 

(0.04) 

Control group 

2.16 

2.14 

-0.02 

.74 

-0.03 

144 

(0.65) 

(0.63) 

(0.04) 

Intervention group 

2.19 

2.30 

0.12* 

.05 

0.18 

85 

(0.66) 

(0.67) 

(0.06) 

Student-staff relationships 

Knows name (SLC staff 0 ) 

All 

0.40 

0.46 

0.06* 

.00 

0.22 

228 

(0.26) 

(0.27) 

(0.01) 

Control group 

0.39 

0.44 

0.05* 

.01 

0.20 

143 

(0.25) 

(0.26) 

(0.01) 

Intervention group 

0.43 

0.50 

0.07* 

.00 

0.25 

85 

(0.28) 

(0.28) 

(0.02) 

Comfort talking (SLC staff 1 ) 

All 

2.50 

2.58 

0.08 

.32 

0.07 

222 

(1.08) 

(1.12) 

(0.05) 

Control group 

2.49 

2.48 

-0.01 

.89 

-0.01 

141 

(1.06) 

(1.10) 

(0.07) 

Intervention group 

2.52 

2.76 

0.24* 

.03 

0.21 

81 

(1.13) 

(1.14) 

(0.09) 

Caring relationships 

All 

3.18 

3.17 

-0.01 

.80 

-0.02 

230 

(0.65) 

(0.73) 

(0.04) 

Control group 

3.16 

3.16 

0.00 

.99 

0.00 

145 

(0.65) 

(0.70) 

(0.05) 

Intervention group 

3.20 

3.17 

-0.03 

.61 

-0.05 

85 

(0.66) 

(0.78) 

(0.06) 

School bonding 

All 

2.91 

2.87 

-0.04 

.22 

-0.07 

230 

(0.59) 

(0.57) 

(0.03) 

Control group 

2.92 

2.87 

-0.06 

.24 

-0.10 

145 

(0.55) 

(0.57) 

(0.03) 

Intervention group 

2.90 

2.87 

-0.02 

.56 

-0.03 

85 

(0.66) 

(0.59) 

(0.04) 

Peer support 

All 

2.57 

2.50 

-0.06 

.17 

-0.10 

229 

(0.67) 

(0.63) 

(0.03) 

Control group 

2.59 

2.54 

-0.05 

.27 

-0.07 

144 

(0.66) 

(0.61) 

(0.04) 

Intervention group 

2.53 

2.44 

-0.09 

.28 

-0.13 

85 

(0.70) 

(0.66) 

(0.07) 

(continued) 
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Table C2. Results of the student surveys conducted 1 week before and 12 weeks 
after participation in a student listening circle, 2015 (continued) 



Means 

Difference between 
pretest and follow-up 


Student outcome measure 

Pretest 

(standard 

deviation) 

Posttest 

(standard 

deviation) 

Difference 3 

(standard 

error) 

p-value 

Standardized 

difference 

Student 

sample 

size 

Competencies 

School improvement efficacy 

All 

2.85 

2.76 

-0.10* 

.03 

-0.19 

229 

(0.50) 

(0.60) 

(0.03) 

Control group 

2.85 

2.75 

-0.10 

.08 

-0.21 

144 

(0.49) 

(0.59) 

(0.04) 

Intervention group 

2.86 

2.77 

-0.09 

.07 

-0.16 

85 

(0.53) 

(0.62) 

(0.05) 

Academic self-efficacy 

All 

3.74 

3.63 

-0.11* 

.01 

-0.14 

229 

(0.82) 

(0.84) 

(0.03) 

Control group 

3.72 

3.62 

-0.09* 

.05 

-0.11 

145 

(0.82) 

(0.85) 

(0.04) 

Intervention group 

3.77 

3.63 

-0.14* 

.01 

-0.17 

84 

(0.84) 

(0.82) 

(0.06) 


* Statistically different from zero at p < .05 based on a two-tailed test. 

a. Differences shown in this column may vary from differences calculated from data in the table because of 
rounding. 

b. SLC staff are staff members who participated in the student listening circle. 

Note: Higher numbers indicate more positive perceptions of school climate and competencies (see table 1 in 
the main text for descriptions of the student outcome measures). P-values are based on a regression model 
that accounted for clustering of observations within schools. 

Source: Authors' analysis of primary data collected for the study through surveys. 
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Appendix D. Student and staff surveys 


This appendix includes the surveys administered to students and staff who participated in 
the study. 

Student survey 

Student Listening Circle Study 
Student Consent Form 

Dear Student, 

You are being asked to participate in a survey that asks questions about you and your expe- 
rience at school. It will take about 30 minutes to answer the questions. 

The survey is voluntary. You do not have to complete this survey. 

The questions we will ask are only about what you think. There are no right or wrong 
answers because this is not a test. 

Taking the survey is up to you; no one will be upset if you don’t sign this paper or if you 
change your mind later. 

Thank you. 

YES, I agree to fill out the Student Listening Circle surveys. 

Print Your First and Last Name: 

Today’s Date: 

After printing your name and today’s date please remove this page from the survey 
booklet and return it to the survey administrator. 
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Dear SLC Student Survey Participant, 


If you are willing to take the survey, please make sure you wrote your name and 
today’s date on the previous sheet and returned it to the survey administrator. That 
page is kept separate from the survey so that your name is not on the survey with your 
answers. Your answers will be kept private! 

You do not have to answer these questions, but we hope you will. Skip any question you 
don’t want to answer. 

Please read every question carefully. Fill in the bubble on only one answer unless the ques- 
tion asks you to mark all answers that apply. 

Please bubble in your responses directly onto the survey using a #2 pencil. 

Thank you for taking the survey! 
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1 . 


What grade are you in now? 

O 6th grade 
O 7th grade 
O 8th grade 

2. Last year , did you attend the same school as now? 

O No 

O Yes 

3. What is your sex? 

O Female 

O Male 

4- Are you of Hispanic or Latino origin? 

O No 
O Yes 

5. Which one of these groups BEST describes you? (Mark only ONE answer.) 

O American Indian or Alaska Native 

O Asian 

O Black or African American 
O Native Hawaiian or Pacific Islander 
O White 

O Mixed (two or more) races 

6. During the past 12 months , how would you describe the grades you mostly received in 
school? 

O Mostly As 
O Mostly A’s and B’s 
O Mostly B’s 
O Mostly B’s and C’s 
O Mostly C’s 
O Mostly C’s and D’s 
O Mostly D’s 
O Mostly F’s 

How strongly do you disagree or agree with the following statements? 

7. I feel like a real part of this school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

8. Sometimes I don’t feel as if I belong at this school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 
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9. I am included in lots of activities at this school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

10. I feel very different from most students here. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

11. I wish I were in a different school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

12. I feel proud to belong to this school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

13. In our school, students are given a chance to help make decisions. 
O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

14. Students in this school have a say in how things work. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

15. Students get to help decide some of the rules in this school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

16. Teachers ask students what they want to learn about. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 
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17. Students help decide how class time is spent. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

18. I enjoy participating at school because I want to have as much say in my school as 
possible. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

19. Students like me can really understand what’s going on with my school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

20. I feel like I have a pretty good understanding of the important issues which confront 
my school. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

21. Students like me have the ability to participate effectively in school activities and 
decisionmaking. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

22. My opinion is important because it could someday make a difference in my school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

23. There are plenty of ways for students like me to have a say in what our school does. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 
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24- It is important to me that I actively participate in student issues. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

25. Most school leaders would listen to me. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

26. Many school activities are important to participate in. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

27. At my school, there is a teacher or some other adult who really cares about me. 

O Not at all true 

O A little true 
O Pretty much true 
O Very much true 

28. At my school, there is a teacher or some other adult who tells me when I do a good job. 
O Not at all true 

O A little true 
O Pretty much true 
O Very much true 

29. At my school, there is a teacher or some other adult who notices when I am not there. 
O Not at all true 

O A little true 
O Pretty much true 
O Very much true 

30. At my school, there is a teacher or some other adult who always wants me to do my 
best. 

O Not at all true 
O A little true 
O Pretty much true 
O Very much true 

31. At my school, there is a teacher or some other adult who listens to me when I have 
something to say. 

O Not at all true 
O A little true 
O Pretty much true 
O Very much true 
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32. At my school, there is a teacher or some other adult who believes that I will be a 
success. 

O Not at all true 
O A little true 
O Pretty much true 
O Very much true 

Think about students your age (not just your closest friends) at your school. Mark how true 
each of the following statements is for you (if unsure, make your best guess). 

33. Students my age really care about what happens to me. 

O Not at all true 

O A little true 
O Pretty much true 
O Very much true 

34- Students my age are there for me whenever I need help. 

O Not at all true 
O A little true 
O Pretty much true 
O Very much true 

35. Students my age can be trusted a lot. 

O Not at all true 

O A little true 
O Pretty much true 
O Very much true 

36. Students my age care about my feelings. 

O Not at all true 

O A little true 
O Pretty much true 
O Very much true 

37. Students my age only think about themselves. 

O Not at all true 

O A little true 
O Pretty much true 
O Very much true 

38. Students my age think bad things about me. 

O Not at all true 

O A little true 
O Pretty much true 
O Very much true 
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For the next questions, please rate how well you can do the following: 

39. How well can you get teachers to help you when you get stuck on homework? 
O Not at all 

O 

O 

O 

O Very well 

40. How well can you study when there are other interesting things to do? 

O Not at all 

O 

O 

O 

O Very well 

41. How well can you study a chapter for a test? 

O Not at all 

O 

O 

O 

O Very well 

42. How well can you succeed in finishing all your homework every day? 

O Not at all 

O 

O 

O 

O Very well 

43. How well can you pay attention during every class? 

O Not at all 

O 

O 

O 

O Very well 

44. How well can you succeed in understanding all subjects in school? 

O Not at all 

O 

O 

O 

O Very well 
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45. How well can you succeed in satisfying your parents with your homework? 

O Not at all 

O 

O 

O 

O Very well 

46. How well can you succeed in passing a test? 

O Not at all 

O 

O 

O 

O Very well 

For each adult at your school listed below, mark one answer for Part A and one answer for 
Part B. 


Part A 


I am certain that she/he knows my name 


47. 


48. 


49. 


Ms. 

O No 
O Yes 

O I don’t know this person 

Mr. 

O No 
O Yes 

O I don’t know this person 

Ms. 

O No 
O Yes 

O I don’t know this person 


<list of names continues until all staff are listed> 
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Part B 


Please rate how comfortable you would be talking to her/him one-on-one about things 
that are bothering you? 

50. Ms. 


O Not at all 
O 
O 
O 

O Very 


51. Mr. 


O Not at all 
O 
O 
O 

O Very 


52. Ms. 


O Not at all 
O 
O 
O 

O Very 


<list of 


names 


continues until all staff are listed> 
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Staff survey 


Student Listening Circle Study 
Consent Form for Adult Participation 

Please complete this page and return it to the SLC Coordinator. 

Adult participation involves: 

Completion of surveys on three different occasions (15 minutes per survey). 

Adult student listening circle invitee, please check the appropriate statement below and 
sign and return this page to the SLC Coordinator. 

YES, I am willing to participate in the Student Listening Circle Study. 

NO, I am not willing to participate in the Student Listening Circle Study. 

Signature: 

Date: 

Name (Please Print): 

Email Address (Used only to send follow-up surveys): 

Please return this page to the survey proctor. 
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Student Listening Circle Study 
Consent to participate 

Student Listening Circle Adult Survey 

Dear SLC Adult Participant, 

• This survey asks questions about things you think, feel, and do. 

• There are no right or wrong answers. 

• You do not have to answer these questions, but we hope you will. Skip any ques 
tion you don’t want to answer. 

• Do not write your name on this paper. We want your answers to stay private! 

• Please read every question carefully. Bubble in only one answer unless the ques 
tion asks you to mark all answers that apply. 

Please bubble in your responses directly onto the survey using a #2 pencil. 

Thank you for taking this survey! 
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1. What is your role(s) at this school? ( Mark all that apply.) 

O Teacher — grade 6 

O Teacher — grade 7 
O Teacher — grade 8 
O Parent of student in school 
O Special education teacher 
O Administrator 

O Prevention staff nurse or health aide 
O Counselor, psychologist 
O Police, resource officer, or safety personnel 
O Paraprofessional, teacher assistant, or instructional aide 
O Other certificated staff (e.g., librarian) 

O Other classified staff (e.g., janitor, secretarial or clerical, food service) 

O Other service provider (e.g., speech, occupational, or physical therapist) 
O Community member not employed by school district 
O Other, please specify: 

How strongly do you disagree or agree with the following statements? 

2. Students share their concerns with me. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

3. Students ask for comfort or support when needed. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

4- Students express their feelings to me. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

5. Students talk with me about their homes and families. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

6. Students talk openly to teachers and staff. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 
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7. Students are motivated to improve the school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

8. Students work well together. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

9. Students collaborate with each other. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

10. Students have a lot to contribute in school improvement decisions. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

11. Students respect the environment and are well-mannered toward others. 
O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

12. Students participate actively in school activities. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

13. Students collaborate with teachers and staff to improve the school. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

14. Students have unique perspectives about barriers to learning. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 
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15. Students offer constructive feedback about the school environment. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

16. This school encourages opportunities for students to decide things like class activities 
or rules. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

17. This school gives all students equal opportunity to participate in numerous extracun 
ricular and enrichment activities. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

18. This school gives students opportunities to “make a difference” by helping other 
people, the school, or the community. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

19. In this school, students are given a chance to help make decisions. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

20. Students are involved in helping to solve school problems. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

21. The principal asks students about their ideas. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 
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22. Students in this school are reliable. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

23. Students in this school can be counted on. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

24- The students in this school have to be closely supervised. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

25. Students are caring toward one another. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

26. Students here are secretive. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

27. Students in this school cheat if they have a chance. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

28. Students in this school can be counted on to do their work. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

29. The students in this school talk freely about their lives outside of school. 
O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 
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30. I believe that I am helping kids become independent learners. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

31. I believe that I am empowering students. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

32. I feel that I am involved in an important program for children. 

O Strongly disagree 

O Disagree 
O Agree 
O Strongly agree 

33. I see students learn. 

O Strongly disagree 
O Disagree 

O Agree 
O Strongly agree 

34- I believe that I have the opportunity to grow by working daily with students. 
O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

35. I perceive that I am making a difference. 

O Strongly disagree 
O Disagree 
O Agree 
O Strongly agree 

Thank you for taking this survey! 
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Notes 


1. California was one of 11 states selected by the U.S. Department of Education to receive 
a Safe and Supportive Schools grant in October 2010. This four-year grant supported 
development of a statewide school climate measurement system and the implementa- 
tion of targeted programmatic interventions to address school safety and bullying, sub- 
stance abuse, positive relationships, other learning supports, and student engagement 
in school. 

2. Students in the intervention group reported greater levels of comfort talking with SLC 
staff 12 weeks after SLC implementation than did students in the control group at the 
.02 significance level, but the difference was no longer statistically significant after an 
adjustment to compensate for the effects of conducting multiple statistical tests. 

3. Although the results in table 4 represent regression-adjusted differences between the 
intervention and control group at a single point in time, the difference score represents 
the intervention/control group difference in change between pretest and follow-up 
because the impact model includes the pretest score as a covariate. 

4. Students in the intervention group (but not in the control group) reported greater 
increases in perceptions of participation in decisionmaking at school and more 
comfort talking with staff participants in the student listening circles 12 weeks after 
the listening circle. These intervention/control group differences were not statistically 
significant after accounting for the increased probability of finding statistically signifi- 
cant differences due to the number of statistical tests conducted. 

5. Dichotomous variables representing each stratification category were included in the 
impact analysis models to account for the unequal random assignment probabilities for 
boys and girls. 

6. In two schools, because of late arrivals of students on the day of the student listening 
circle, a replacement student was randomly selected to participate when eight inter- 
vention group students were already available. This resulted in nine SLC participants 
— eight intervention group students and one replacement student in these schools. 

7. Because students in the replacement pool were randomly selected, they could have 
been transferred to the intervention group for data analyses without biasing compar- 
isons between groups. Nevertheless, replacement students were kept in their original 
random assignment group as a conservative approach. The results were unchanged 
when replacement students were treated as intervention group students in the analysis. 

8. To ease the survey burden, the grid system listed a maximum of 44 school staff. If this 
number was insufficient to accommodate all school staff, the survey listed all SLC 
adult participants who were school staff and randomly selected staff members until 44 
slots on the questionnaire were filled. 

9. The scale developed by Peterson et al. (2011) includes five level-of-agreement response 
options. For this study, the neutral option (neither agree nor disagree) was dropped, 
resulting in four response options. 

10. The California School Climate, Health, and Learning Survey and Alaska School 
Climate and Connectedness Survey student involvement items have not previously 
been combined to form a summary measure. Therefore, evidence regarding reliabil- 
ity of this new seven-item scale is not available. Factor analyses conducted to deter- 
mine whether these seven items measure a single dimension of student opportunities 
for meaningful involvement indicated that these seven items do measure a single 
dimension. 
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